1. Field of the Invention
The present invention relates to an image processing apparatus, an image processing method and an image processing program. More particularly, the present invention relates to an image processing apparatus for carrying out NR (noise reduction) processing in addition to execution of SR (super-resolution) processing on an image in order to raise the resolution of the image, relates to an image processing method to be adopted by the apparatus and relates to an image processing program implementing the method.
2. Description of the Related Art
As a technique for generating an image with a high resolution from an image with a low resolution, there is known a technique for carrying out SR processing which is processing for generating an image with a high resolution from an image with a low resolution.
As a technique for carrying out the SR processing, for example, there is known a reconfiguration-type super-resolution technique for inferring an ideal image with a high resolution on the basis of a photographing image of an image with a low resolution by making use of parameters derived as parameters showing photographing conditions such as “blurring caused by lens and atmosphere scatterings,” “motions of the photographing object and the entire camera” and “sampling by the imaging device.”
An existing technology disclosed as a technology underlying a technique for carrying out the super-resolution processing is described in documents such as Patent Document 1 which is Japanese Patent Laid-open No. 2008-140012.
An outline of the procedure of the reconfiguration-type super-resolution technique is described as follows:
(1) An image photographing model considering photographing conditions such as blurs, motions and samplings is expressed by a mathematical formula.
(2) A cost computation formula is found from the image photographing model expressed by the mathematical formula. In the process to find the cost formula, a regularization term such as an advance confirmation term may also be added by adoption of the Bayesian theory in some cases.(3) An image minimizing the cost is found.
A technique for finding an image having a high resolution is implemented by carrying out these kinds of processing. Even though the high-resolution image obtained as a result of the reconfiguration-type super-resolution technique is dependent on the input image, the super-resolution effect (or the resolution restoration effect) is big.
FIG. 1 is a block diagram showing a typical configuration of an SR processing circuit for carrying out the super-resolution processing. That is to say, FIG. 1 is a block diagram showing a typical configuration of a super-resolution processing apparatus 10.
The super-resolution processing apparatus 10 carries out processing described as follows. An up-sampling section 11 receives a low-resolution input image (gn) 31 to be subjected to processing to raise the resolution of the image. The up-sampling section 11 carries out up-sampling processing which is a resolution conversion process to raise the resolution of the low-resolution input image (gn) 31. That is to say, the up-sampling section 11 carries out processing to adjust the resolution of the low-resolution input image (gn) 31 to the resolution of an SR (super resolution)-processed image (fn) 32 which is a high-resolution image to be eventually generated. For example, the up-sampling section 11 carries out enlargement processing to divide every pixel of the low-resolution input image (gn) 31 into a plurality of pixels of the SR-processed image (fn) 32. In the following description, the SR-processed image is also referred to as an SR (super resolution) processing result image.
A motion-inference/motion-compensated-image generation section 12 detects the magnitude of a motion from a high-resolution image (fn-1) generated in the processing carried out on an immediately preceding frame to a high-resolution up-sampled image obtained as a result of the up-sampling processing carried out by the up-sampling section 11 on the low-resolution input image (gn) 31. To put it concretely, the motion-inference/motion-compensated-image generation section 12 computes a motion vector between the high-resolution image (fn-1) and the high-resolution up-sampled image. Then, by making use of the computed motion vector, the motion-inference/motion-compensated-image generation section 12 carries out MC (motion compensation) processing on the high-resolution image (fn-1). To put it in detail, the motion-inference/motion-compensated-image generation section 12 carries out the MC processing on the high-resolution image (fn-1) in order to generate a motion compensation result image setting the photographing object at the same position as the photographing object in the high-resolution up-sampled image obtained as a result of the up-sampling processing carried out by the up-sampling section 11 on the low-resolution input image (gn) 31. In the following descriptions, the motion compensation result image is also referred to as a motion-compensated image.
A motion determination section 13 compares the motion-compensated high-resolution image obtained as a result of the MC processing with a high-resolution up-sampled image obtained as a result of the up-sampling processing carried out by the up-sampling section 11 on the low-resolution input image (gn) 31 in order to detect an area to which the MC processing may not be applied well. When the photographing object itself is moving for example, an unsuccessful motion-compensation area is generated.
The motion determination section 13 generates motion area information referred to as an α map [0:1] for distinguishing a successful motion-compensation area from an unsuccessful motion-compensation area. A successful motion-compensation area is an area in which a motion compensation result image generated by the motion-inference/motion-compensated-image generation section 12 on the basis of the high-resolution image fn-1 of the immediately preceding frame sets the photographing object at the same position as the photographing object in the high-resolution up-sampled image obtained as a result of the up-sampling processing carried out by the up-sampling section 11 on the low-resolution input image (gn) 31. On the other hand, an unsuccessful motion-compensation area is an area in which the motion compensation result image generated on the basis of the high-resolution image fn-1 of the immediately preceding frame does not set the photographing object at the same position as the photographing object in the high-resolution up-sampled image obtained as a result of the up-sampling processing carried out by the up-sampling section 11 on the low-resolution input image (gn) 31. The motion area information referred to as an α map [0:1] is a map having a value varying in the range 1 to 0 in accordance with the degree of confidence for the successful motion-compensation area and the unsuccessful motion-compensation area. In a simple application for example, the value of the α map [0:1] for an area is set at 1 in order to indicate that the area is a successful motion-compensation area or set at 0 in order to indicate that the area is an unsuccessful motion-compensation area.
A blending processing section 14 receives a motion compensation result image generated by the motion-inference/motion-compensated-image generation section 12 on the basis of the high-resolution image fn-1 of the immediately preceding frame, a high-resolution up-sampled image obtained as a result of the up-sampling processing carried out by the up-sampling section 11 on a low-resolution input image (gn) 31 and the motion area information referred to as an α map [0:1] generated by the motion determination section 13.
The blending processing section 14 makes use of the received inputs described above to output a blended image obtained as a result of blending processing carried out on the basis of an equation given as follows:Blended image=(1−α)(high-resolution up-sampled image)+α(motion compensation result image)
In the above equation, reference symbol a denotes the aforementioned α map [0:1] having a value in the range 0 to 1. The value of the α map [0:1] is a blending coefficient α of the motion compensation result image. The blending coefficient α is used as a weight assigned to the motion compensation result image in the blending processing. By carrying out the blending processing, it is possible to generate a blended image by making use of an increased blending coefficient α as a weight assigned to the motion compensation result image for a successful motion-compensation area but a decreased blending coefficient α as a weight assigned to the motion compensation result image for an unsuccessful motion-compensation area.
In the following description, a blending ratio is defined as the ratio α/(1−α) in order to distinguish the blending ratio from the blending coefficient α.
A blur addition section 15 receives the blended image generated by the blending processing section 14 and carries out simulation processing for the blended image to simulate deteriorations of the spatial resolution. For example, the blur addition section 15 carries out convolution to the image by taking a PSF (point spread function) measured in advance as a filter.
A down-sampling section 16 carries out down-sampling processing to lower the resolution of the high-resolution image to the same resolution as the low-resolution input image (gn) 31. Then, a subtractor 17 computes a difference between an image output by the down-sampling section 16 and the low-resolution input image (gn) 31 for every pixel. The difference is subsequently subjected to up-sampling processing carried out by an up-sampling section 18. Then, an inverse blur addition section 19 carries out processing inverse to blur addition processing. The operation of the inverse blur addition processing is the same as processing to compute a correlation with the PSF (point spread function) used in the blur addition section 15.
A multiplier 20 carries out an operation to multiply the image output by the inverse blur addition section 19 by an SR feedback coefficient γSR set in advance in order to generate a product and supplies the product to an adder 21. Then, the adder 21 carries out an operation to add the product to the blended image generated by the blending processing section 14 in order to generate the output image of the super-resolution processing apparatus 10.
The image generated by the adder 21 as the output of the super-resolution processing apparatus 10 is an SR (super resolution) processing-result image (fn) 32 which is an image obtained as a result of raising the resolution of the low-resolution input image (gn) 31.
The processing carried out by the super-resolution processing apparatus 10 shown in FIG. 1 can be expressed by the following equation:fnSR=(Wnfn-1SR)′+γSRHTDT(gn−DH(Wnfn-1SR)′)  (1)
It is to be noted that a variety of reference symbols used in Eq. (1) given above denote parameters described as follows.
n: Frame number. For example, frames (n−1) and n are consecutive frames of a moving image.
gn: Input image (a low-resolution image of frame n)
fnSR: Super-resolution image of frame n (an image obtained as a result of super-resolution processing for frame n)
fn-1SR: Super-resolution image of frame n−1 (an image obtained as a result of super-resolution processing for frame n−1)
Wn: Information on a frame motion from frame (n−1) to frame n. This information is expressed by typically a motion vector or a matrix.
H: Blur addition processing (expressed by a blur filter matrix)
D: Down-sampling processing (expressed by a down-sampling processing matrix)
(Wnfn-1SR)′: Blended image output by the blending processing section
γSR: SR feedback coefficient
HT: Transposed matrix of matrix H
DT: Transposed matrix of matrix D
In accordance with a super resolution technique adopted by the super-resolution processing apparatus 10 shown in FIG. 1, the following processing is carried out effectively.
(a) On the basis of a folding-back component of an input image, high-frequency components are inferred. In the following description, the folding-back component is also referred to as an aliasing. The high-frequency components are components each having a frequency at least equal to the Nyquist frequency.(b) Folding-back components within low-frequency components are eliminated in order to restore high-frequency components. The low-frequency components are components each having a frequency not exceeding the Nyquist frequency.
If a magnifying power specified to generate a super-resolution image does not match the magnifying power of an alias component caused by a resolution set from the beginning prior to contraction of the image, however, there is raised a problem that an alias is left in the image as before. In addition, an alias component not corrected because the image has been determined to be the image of a moving body is also left inevitably in the image as noises. As a result, there is raised a problem that noises caused by an alias component are undesirably left.